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ADAPTIVE FACIAL RECOGNITION SYSTEM AND METHOD 
BACKGROUND OF THE INVENTION 

1. Technical Field 

The present invention relates to facial recognition, and more particularly relates to 
5 an adaptive system for detection and tracking of faces. 



2. Related Art 

As electronic commerce and information becomes more prevalent in our society, 
security issues have become an ongoing and important challenge. Such challenges exist 
both in peoples' business and in home environments. For instance, in business 

10 environments, security is required for transactions such as banking at an ATM, 

purchasing goods with a credit card, or downloading secure data from the Internet. 
Similarly, in some households it may be desirable to prevent children from viewing 
undesirable material on the internet or TV. In order to provide security in such 
environments, the particular systems need to correctly establish the identity of the 

1 5 participants. A traditional method of establishing identity is through the use of 

passwords, such as a PIN number. Unfortunately, because passwords can be forgotten, 
stolen, disseminated, etc., they provide only a limited form of security and can be readily 
defeated. 

In order to overcome such limitations, recent security developments have focused 
20 on "biometrics," which is a term that describes automated methods of establishing a 
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person's identity from their unique physiological or behavioral characteristics. 
Fingerprinting, retina scans and handwriting recognition are all examples of biometrics 
that can or have been used to establish identity. Unfortunately, most security systems 
that use biometric applications not only require specialized hardware (e.g., a retinal 
5 scanner), but may also be seen as intrusive to one's personal privacy. 

One form of biometric security that is relatively non-intrusive involves facial 
recognition, in which an image of a person's face can be digitally compared to a 
previously stored image. An example of such a system is disclosed in U.S. Reissue 
Patent No. 36,041, entitled, "FACE RECOGNITION SYSTEM," issued to Turk et al, 

10 and is hereby incorporated by reference. As disclosed, a stored reference face, which 
comprises facial images characterized as a set of eigenvectors or "eigenfaces," can be 
used to identify or authenticate an individual. 

One of the challenges of the above-mentioned face recognition system is the need 
to perform "off-line" training, which involves gathering multiple face images each time a 

15 new individual is added to the database of faces. Unfortunately, such a process is often 
too complicated, time-consuming, costly or impractical. For instance, in a relatively 
limited-scale environment, such as home or small office, people may lack the technical 
know-how or desire to set up and implement a training system. In a large-scale 
environment, such as a bank, it may be impractical to bring in each customer for video 

20 imaging so that they can be recognized for future ATM visits. Accordingly, a need exists 
for a face recognition system in which offline training is not required. 
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SUMMARY OF THE INVENTION 



The present invention addresses the above-mentioned problems by providing an 
adaptive face recognition system and method that does not require off-line training. In a 
first aspect, the invention provides an adaptive face recognition system, comprising: a 
5 database configured to store a plurality of face classes; an image capturing system for 
capturing images; a detection system, wherein the detection system detects face images 
by comparing captured images with a generic face image; a search engine for 
determining if a detected face image belongs to one of a plurality of known face class; 
and a system for generating a new face class for the detected face image if the search 
10 engine determines that the detected face image does not belong to one of the known face 
classes. 

In a second aspect, the invention provides a method for performing adaptive face 
recognition, comprising the steps of: capturing a stream of image data; identifying a face 
image from the stream of image data by comparing the image data to a generic face 

15 image; searching a database of face classes to determine if the detected face image 

belongs to one of a plurality of known face classes; if the detected face image belongs to 
one of the known face classes, adding the detected face image to the face class that owns 
the face image; and if the face image does not belong to one of the known face classes, 
creating a new face class with the face image. 

20 In a third aspect, the invention provides a program product stored on a recordable 

medium for performing adaptive face recognition, that when executed, comprises: a 
system for receiving image data; a system for detecting a face image from the received 
images by comparing the image data to a generic face image; a system for searching a 
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database of face classes to determine if a detected face image belongs to one of a plurality 
of known face classes; a system for adding the detected face image to an associated face 
class if the detected face image belongs to one of the known face classes; and a system 
for creating a new face class with the detected face image if the detected face image does 
5 not belong to one of the known face classes. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The preferred exemplary embodiment of the present invention will hereinafter be 
described in conjunction with the appended drawings, where like designations denote like 
elements, and: 

10 Figure 1 depicts a block diagram of an adaptive face recognition system in 

accordance with a preferred embodiment of the present invention. 

Figure 2 depicts an exemplary database record for the system of Figure 1. 



DETAILED DESCRIPTION OF THE INVENTION 

Overview 

1 5 The present invention provides a system that gradually learns different faces from 

ongoing "image collection events." An image collection event may occur anytime an 
individual's face is presented and detected by the system. Rather than pre-train the 
system with new faces off-line, the system generates a new face class each time a new 
face is presented, and associates the individual's face images presented at later times to 
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the same class. The system can be used for any application in which identity of an 
individual is required. Examples include, but are not limited to, intruder detection, enter- 
exit event detection, user profiling, human-machine interaction applications, ATM's, 
smart card access, etc. The system automatically updates user face feature 
5 representations on-line so that the system gradually learns appearance changes of an 
individual over time in a cost effective manner. 

The system implements what is referred to herein as an "adaptive eigenface," 
which gradually learns the specific traits of an individual's face as new images are 
presented to the system. In a live scene, once a face is detected, a face tracking system 

10 may be utilized to not only continuously follow the same face, but also automatically add 
acceptable face images of the person for on-line adaptive training. If the face is 
unrecognized, a new class for this person can be created. If a face image is recognized as 
belonging to a known class, acceptable face images can be sequentially added for 
adaptive training to further strengthen its class representation. 

1 5 Face detection (which occurs prior to face recognition) is accomplished by 

comparing an obtained image with a generic face. The generic face may be created off- 
line ahead of time using a large number of face images from various people and is 
represented by a set of eigentemplates. 

Preferred Embodiment 

20 Referring now to the figures, Figure 1 depicts an adaptive face recognition system 

10, which adaptively learns and recognizes faces, such as that of individual 12. The 
system 10 operates by first collecting image data with a video camera 16 or similar 
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device. Image data may comprise any type of image information, including streaming 
video, digital pictures, digital video, analog video, etc. Once captured, image data is then 
loaded into system 10 via image input 20. Image data is then passed to face detection 
system 22, which determines if a face image exists in the inputted image data. Faces are 
5 detected by comparing information in an inputted image with a generic face 28 stored in 
a database 26. Generic face 28 may be represented by a set of eigentemplates trained off- 
line by generic face off-line training system 29. The processes for creating a generic face 
are readily known in the art. 

When a face image is detected by face detection system 22, search engine 24 

10 searches the known face classes 30 of database 26 to determine if the face image belongs 
to one of the known classes (i.e., the face is recognized). The search engine 24 may 
utilize a distance criterion such as that used by an eigenface method to determine if a 
detected face image belongs to a known class. Such eigenface methods, which are well 
known in the art, represent each face class by a subspace that is spanned by a small 

1 5 number of principal components extracted from a set of face data containing face images 
of the same class. The eigenface method employs a distance criterion that considers the 
distance to the face subspace and the distance within the face subspace for determining if 
a candidate vector belongs to the same face class or not. 

If it is determined that the face image belongs to one of the known classes (i.e., an 

20 owning class), it is characterized as known 32. If the face image is known 32, adaptive 
training system 38 may be used to sequentially update the owning class with the detected 
image. A selection mechanism 40 may be incorporated to select only acceptable images 
for training. Such on-line training may be achieved by applying a sequential eigen 
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decomposition. Known methods for accomplishing this include the power method and/or 
orthogonal iterations that update the eigentemplates when new data vector information is 
presented to the system. As more and more faces for the same person are added, the class 
representation is improved, thereby improving the detection rate of the known class. 
5 Using this process, only a relatively small number of iterations will be required to either 
(1) adapt the generic face eigentemplates to those for a specific face, or (2) generate a 
new face representation from scratch. 

If the search engine 24 determines that a detected face image does not belong to 
one of the known classes 30 (i.e., it is unknown 34), then a new class for the 

10 unrecognized face image is generated by new class generation system 36. The new class 
with the detected image is added to database 26. The newly created class therefore 
becomes one of the known classes for future searches by search engine 24. Thus, the 
process of off-line training is eliminated in favor of on-line sequential training. To 
enhance the training process, a face tracking system 18 can be utilized to control the 

15 video recorder 16 to lock onto and follow an individual's face. Thus, a tracking event 
can occur in which numerous different facial images of an individual can be collected 
during a single event. Overall, the computational costs for sequential updating is far less 
than initial off-line training, which generally takes a large number of data vectors to build 
a sufficient class representation. 

20 In addition to detecting and adaptively training facial recognition, system 10 

provides a control system 44 for controlling access to, and use of, external applications 
14 based on privileges set in database 26. For instance, if a known individual 12 was 
seeking access to a web browser application, the control system 44 could determine the 
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privileges allocated to individual 12 for the web browser. Specifically, when search 
engine 24 recognizes an individual as belonging to a known class, an identifier 42 for that 
person can be communicated to control system 44, which will check the associated face 
class for the recognized individual 12 to determine the individual's privilege level for the 
5 application 14. 

Control system 44 further includes an administrative interface 46 that allows an 
administrator to, among other things, preset privilege levels for each application. 
Additionally, administrative interface 46 provides access to reports 47 generated by 
control system 44 that show what applications were used, when they were used, and 

10 which identified person used which application. Thus, for example, a parent could 
determine when a child was surfing the web, watching television, etc. 

Referring now to Figure 2, an exemplary database record 31 for face class 1 is 
shown. Record 31 includes image data 33 collected during a tracking event. The image 
data 33 may comprise, for example, eigentemplates that include frontal views, side views, 

1 5 etc. It should be understood that any system for storing feature representations or 
signatures could be utilized. Also included in record 31 are exemplary application 
privilege settings 35 for face class 1. In this case, the settings include a name "Junior" 
for the face class and a label "Child." The label can be used to classify groups of similar 
individuals represented in known face classes. Other labels could include, for example, 

20 adult, employee, administrator, owner, etc. Each label could have a set of default 

privilege settings. In this example, the label "Child," has several default settings that 
dictate the privileges for the individuals having this label. Specifically, for the three 
applications "TV," "web," and "telephone," the Child label dictates that Junior has 
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privilege settings of "PG," "G," and "local." Labels and associated default settings may 
be set via the administrative interface 46 by an administrator (e.g., a parent). 

Accordingly, when Junior attempts to use of the three listed applications, his face 
image will be detected and recognized as belonging to known class 1. Control system 44 
5 can communicate Junior's privilege settings for the application Junior seeks to use. The 
application can then be configured to limit Junior's use to the prescribed settings. It 
should be understood that the described method of implementing control system 44 and 
the associated settings in the known face classes is for exemplary purposes only and 
should not be considered limiting. 

10 It is understood that the systems, functions, mechanisms, and modules described 

herein can be implemented in hardware, software, or a combination of hardware and 
software. They may be implemented by any type of computer system or other apparatus 
adapted for carrying out the methods described herein. A typical combination of 
hardware and software could be a general-purpose computer system with a computer 

15 program that, when loaded and executed, controls the computer system such that it 
carries out the methods described herein. Alternatively, a specific use computer, 
containing specialized hardware for carrying out one or more of the functional tasks of 
the invention could be utilized. The present invention can also be embedded in a 
computer program product, which comprises all the features enabling the implementation 

20 of the methods and functions described herein, and which - when loaded in a computer 
system - is able to carry out these methods and functions. Computer program, software 
program, program, program product, or software, in the present context mean any 
expression, in any language, code or notation, of a set of instructions intended to cause a 
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system having an information processing capability to perform a particular function 
either directly or after either or both of the following: (a) conversion to another language, 
code or notation; and/or (b) reproduction in a different material form. 

The foregoing description of the preferred embodiments of the invention have 
been presented for purposes of illustration and description. They are not intended to be 
exhaustive or to limit the invention to the precise form disclosed, and obviously many 
modifications and variations are possible in light of the above teachings. Such 
modifications and variations that are apparent to a person skilled in the art are intended to 
be included within the scope of this invention as defined by the accompanying claims. 
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